optimize(interleave): implement interleave for FixedSizeList/Map type#10046
optimize(interleave): implement interleave for FixedSizeList/Map type#10046mapleFU wants to merge 4 commits into
Conversation
| ("list_view<i64>(0.0,0.0,20)", &list_view_i64_no_nulls), | ||
| ("list_view_overlapping<i64>(80x,20)", &list_view_overlapping), | ||
| ("fixed_size_list<i64,5>(0.0,0.0)", &fsl_i64), | ||
| ("fixed_size_list<i64,5>(0.1,0.1)", &fsl_i64_nulls), |
There was a problem hiding this comment.
Maybe I can also support large fsl, like vector<1024, f32>
There was a problem hiding this comment.
I don't know should I extract benchmark to a separate pr ( containing this and concat benchmark?)
|
Will refactor since #10025 is merged |
| Ok(Arc::new(array)) | ||
| } | ||
|
|
||
| fn interleave_map( |
There was a problem hiding this comment.
map child is always complex type, so doesn't call the primitive fn
There was a problem hiding this comment.
Pull request overview
This PR extends arrow-select’s interleave kernel to natively support FixedSizeList and Map arrays (previously falling back to the generic implementation), and adds benchmark inputs to measure the new paths.
Changes:
- Add
interleaveimplementations forFixedSizeListandMapdata types inarrow-select - Refactor list primitive-child interleave to use the shared
ListLikeArrayabstraction - Add benchmark data generators and new benchmark cases for
FixedSizeList<i64>andMap<utf8, i64>
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
arrow/src/util/bench_util.rs |
Adds benchmark data generators for fixed-size lists and string-keyed maps |
arrow/benches/interleave_kernels.rs |
Adds new benchmark cases for FixedSizeList and Map interleave |
arrow-select/src/interleave.rs |
Implements native interleave for FixedSizeList/Map and refactors list-child interleave via ListLikeArray |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| let mut builder = builder::MapBuilder::new( | ||
| None, | ||
| builder::StringBuilder::new(), | ||
| builder::PrimitiveBuilder::<T>::new(), | ||
| ); |
| let mut rng = seedable_rng(); | ||
| let list_size_usize = list_size as usize; | ||
| let values: PrimitiveArray<T> = (0..size * list_size_usize) | ||
| .map(|_| { |
Which issue does this PR close?
Rationale for this change
Implement interleave for Map/FSL type.
Note: This might a slow implementation for FSL. For map it's a struct type, and at would have two childs. So even if it's
Map<i32, i32>, it wouldn't be slower than before.For FSL, if it's fixed sized list with short i32 list, it may hit the problem in #10025 , making them use same code to handle list is better.
What changes are included in this PR?
Implement interleave for Map/FSL type.
Are these changes tested?
Are there any user-facing changes?
No